Repository navigation
feat(api): serve Prometheus metrics and add per-client rate limiting - #49
Merged
Merged
Conversation
Set up the foundation package structure for the HTTP API server that will serve the management CLI and web dashboard. Files created: - doc.go: Package documentation with endpoint overview - config.go: Config struct with DefaultConfig(), Validate(), and builder methods - errors.go: Package-specific error definitions - types.go: JSON request/response type definitions - routes.go: Route registration with /health endpoint - server.go: Server struct with Start/Stop lifecycle - server_test.go: Comprehensive unit tests (11 tests) The API server follows existing GoClusterFS patterns: - Config pattern from internal/cache - Manager lifecycle pattern from internal/journal - Dependency injection (no global singletons) All tests pass (11/11). Task: #5588
- Add /api/v1/ catch-all handler for versioned endpoints - Add /metrics endpoint for Prometheus scraping - Add /api/v1/status endpoint returning server status - Add /api/v1/stats/* placeholder for component stats - Add method validation (405 Method Not Allowed) - Add route tests for all endpoint groups Task: #5591
- Add LoggingMiddleware that logs method, path, status, duration - Add AuthMiddleware with Bearer token validation and skip paths - Add CORSMiddleware with configurable origins and preflight support - Integrate middleware chain into server (logging -> CORS -> auth) - Add comprehensive middleware tests (11 new tests) Task: #5592
- Add HealthStatus constants (ok, degraded, unhealthy) - Add ComponentHealth for individual health checks - Enhance StatusResponse with capacity, cluster, mount state - Add CacheStatsResponse matching internal/cache.Stats - Add JournalStatsResponse matching internal/journal.JournalStats - Add IOStatsResponse and StatsOverviewResponse - Add comprehensive type marshaling tests Task: #5593
- Add HealthChecker interface for component health reporting - Add AggregateHealth to combine multiple health check results - Implement priority: unhealthy > degraded > ok - Return 503 for unhealthy status, 200 for ok/degraded - Add RegisterHealthChecker to Server for dependency injection - Add comprehensive health endpoint tests Task: #5594
…y info - Add version.go with build-time ldflags support (Version, GitCommit, BuildTime) - Add StatusProvider interface for filesystem status injection - Enhance handleStatus with build info and optional capacity - Add status endpoint tests with mock provider Task: #5595
- Add fsStatusProvider to expose filesystem status via API - Start API server on :8080 after FUSE mount succeeds - Register health checker and status provider - Add graceful shutdown in signal handler and exit cleanup Task: #5596
- Add -api flag to enable/disable API server (default: enabled) - Add -api-addr for listen address (default: :8080) - Add -api-token for Bearer token authentication - Add -api-cors for comma-separated CORS allowed origins - Wire flags to API server configuration via fluent API Task: #5597
Add thread-safe accessor method to expose snapshot registry for API endpoints. Uses RLock since we're only reading the pointer and the Registry itself manages its own concurrent access with internal mutex. Task: #5598
Add unified statistics collection and aggregation package: - stats.go: Core types (Snapshot, CacheStats, JournalStats, etc.) and provider interfaces for component decoupling - collector.go: Collector with TTL-based caching to aggregate statistics from multiple providers - cache.go: Cache state utilities (CacheInfo, CachedSnapshot) - errors.go: Package-specific error definitions - doc.go: Package documentation - stats_test.go: Comprehensive unit tests (16 tests) The collector uses a pull model where providers are queried on demand, with configurable cache TTL to avoid expensive repeated queries. Preset configs available for different use cases: DefaultConfig (1s TTL), HighFrequencyConfig (100ms), LowOverheadConfig (10s). Task: #5599
Add comprehensive statistics types for TRIM operations, Sparse File Bitmap allocation, and cluster node information: - TrimStats: TRIM/discard operation metrics - SFBStats: block allocation and fragmentation stats - ClusterInfo: node role, state, and peer information - PeerInfo: individual peer node details Also add provider interfaces and Collector support for all new types. Task: #5600
…sfb, cluster - Add StatsProvider interface for statistics collection - Implement /api/v1/stats endpoints: - /stats (overview with cache and journal) - /stats/cache (inode, extent, negative cache hit rates) - /stats/journal (transactions, space, checkpoints) - /stats/trim (TRIM operations and bytes) - /stats/lock (lock hint cache statistics) - /stats/sfb (block allocation statistics) - /stats/cluster (node membership and peers) - /stats/all (combined snapshot of all stats) - Add response types for all stats endpoints - Add comprehensive tests for stats handlers Task: #5602, #5603, #5604, #5605, #5606, #5607
CI fails on unformatted files. Whitespace and import order only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t a token The API listened on :8080 on every interface with no auth, so any host that could reach the node could read cluster state. -api-addr now defaults to 127.0.0.1:8080. Mount exits before touching the device when -api-addr can be reached from other hosts and -api-token is empty. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Broadcast is on by default and its secret is empty by default, so most clusters accept unauthenticated invalidation and peer messages without knowing it. Mount now logs a WARNING line before it opens the device. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
internal/api kept its own Version, GitCommit and BuildTime, which scripts/dist.sh never set, so /api/v1/status always said "dev". GetBuildInfo now reads internal/version, and goclusterfs-mount -version prints the same string as goclusterfs-fsck -version. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
golangci-lint flagged mockFilesystemProvider as unused. Use it in a test of SetFilesystemProvider instead of deleting it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Token bucket per client IP, off by default (Config.RateLimitEnabled). Requests over the limit get 429 with Retry-After: 1. Recovered from uncommitted work in the user's checkout. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A custom collector reads the StatsProvider on each scrape and exports cache, journal, TRIM, lock, SFB and cluster metrics plus Go runtime metrics. Adds github.com/prometheus/client_golang v1.23.2, which raises golang.org/x/sys from v0.28.0 to v0.35.0. Recovered from uncommitted work in the user's checkout; prometheus.go was gofmt'd. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
StatsProvider gains ClusterNodes and ClusterLocks. The handlers answer 503 without a provider and 404 when the provider has no cluster data. Recovered from uncommitted work in the user's checkout. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
gfsctl wraps the API: status, health, stats, cluster, snapshot, admin, watch, config and shell completion. It reads host, port and token from flags, GFSCTL_* variables or ~/.config/gfsctl/config.yaml. Adds spf13/cobra v1.10.2 and gopkg.in/yaml.v2 v2.4.0. The snapshot and admin commands call /api/v1/snapshots and /api/v1/admin/*, which the server does not have yet. Recovered from uncommitted work in the user's checkout. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Vite + React 19 + Tailwind, with pages for the dashboard, cluster, statistics, snapshots and settings. `npm run dev` serves it on :5173 and proxies /api, /health and /metrics to localhost:8080. Nothing serves the built files yet. Source only: node_modules and dist stay ignored by web/dashboard/.gitignore. The lockfile is the user's from 2025-12-31. Recovered from untracked work in the user's checkout. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
gfsctl kept its own Version, GitCommit and BuildTime, a third copy next to internal/version and the one internal/api dropped. It now uses internal/version like goclusterfs-mount and goclusterfs-fsck. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
prometheus.NewGoCollector and NewProcessCollector are deprecated, and staticcheck (SA1019) fails CI's lint job on them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
allocateImage opens an existing file, so the refusal test failed before it ran the mount. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When the API port is taken, mount logged a second warning at unmount
("api: server not started"). Every mount after the first on a host
hits this.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…c8b883 # Conflicts: # cmd/mount/main.go
…o claude/land-api-work-uncommitted
main's tree equals #48's last head, already merged in the parent commit.
…uncommitted # Conflicts: # internal/api/middleware.go # internal/api/server.go
…wn PR They depend on a stats provider and on snapshot and admin endpoints that the server does not have yet. This PR keeps Prometheus metrics and rate limiting. The removed parts are restored unchanged in a stacked draft PR. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The limiter took the client IP from X-Forwarded-For or X-Real-IP, so any client could pick a fresh bucket per request. It now uses the connection's address only. Its cleanup goroutine ran forever; Server.Stop now ends it, also for a server that never started. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Requests per second per client IP, with bursts of twice that. 0, the default, keeps rate limiting off. A negative value stops mount before it opens the device. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mattmattox
marked this pull request as ready for review
November 10, 2026 20:24
Review of #49 found that the limiter, which runs before auth, kept one bucket per source IP for 10-15 minutes. An IPv6 /64 or a local user on 127.0.0.0/8 could grow it to gigabytes. Now: - A bucket is dropped once it has been idle long enough to refill (burst/rate, at most an hour), swept every 1-60 s. - At most 65536 clients have buckets; new clients past that get 429. - IPv6 clients share a bucket per /64; mapped IPv4 counts as IPv4. Also: - Validate refuses NaN, infinite and above-1e6 rates. - Retry-After is ceil(1/rate), not always 1. - /health and /metrics skip the limiter, so probes and scrapers that share 127.0.0.1 never get 429. - promhttp serves at most 2 scrapes at once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Dropping the server without Stop leaked the limiter's cleanup goroutine. -api-rate-limit now gets a clear error for NaN, infinite or huge values instead of an overflowed burst. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nt cap Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
At 100 requests/s a token refills every 10 ms, so on a loaded host the 4th request often found a fresh token. 1 request/s keeps the bucket empty for the test's duration. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mattmattox
added a commit
that referenced
this pull request
Nov 10, 2026
main's tree equals #49's last head, already merged in the parent commit.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The finished part of the user's uncommitted API work: Prometheus
/metricsand per-client rate limiting. The unfinished parts (gfsctl, the dashboard and the cluster endpoints) moved unchanged to draft #52. They are removed here with a normal commit (2505935), so nothing was force-pushed./metricsusesclient_golangv1.23.2, the user's version. It exportsgoclusterfs_upplus Go runtime and process metrics. The filesystem metrics appear once mount gives the API a stats provider, which nothing does yet.client_golangraisesgolang.org/x/sysfrom v0.28.0 to v0.35.0.-api-rate-limit Nsets N requests per second with bursts of 2N. Over the limit, the API answers 429.X-Forwarded-FororX-Real-IP, so a client could pick a new bucket per request. It now uses the connection's address.Server.Stopnow ends it.Retry-Afteris ceil(1/rate);/healthand/metricsskip the limiter; a failed API start no longer leaks the cleanup goroutine.prometheus.gomoved off the deprecatedNewGoCollectorandNewProcessCollector, and serves at most 2 scrapes at once.Behind a proxy, every client shares the proxy's limit. The docs say so.
Results (head 2094c0c):
go mod tidy -diff, golangci-lint (0 issues), parity andgo test -race ./internal/... ./cmd/...(20 packages) pass. Integration-short: 97 pass, 18 skip, 1 fail. The failure isTestMultiNodeConcurrentDirectoryOps(a concurrent create got EIO from an interrupted lock wait), the known flake that fix(lock): claim locks through per-node slots and cache released holds #29 fixes; it touches no API code. 5382141, which differs only in one test's rate, had 98 pass, 0 fail.TestRateLimitMiddleware_Enabledran at 100 requests/s, so on a loaded host the bucket refilled mid-test. It now runs at 1/s.🤖 Generated with Claude Code